Make the supported examples prove behaviour, not existence (#104) - #446
Conversation
The acceptance suite asserted that a command exited and that some files
existed. A badly broken runtime satisfies that, and one did: 04_conditions
ran *both* arms of its conditional, and the test asserted both artifacts,
pinning the bug in place.
04_conditions now expresses two genuinely exclusive branches converging on a
join. `handle_short` was never a branch -- it was the last link of a
dependency chain, so the default path fell through into it. The long arm's
`on_success` jumps the short arm; the short arm is reached only by `on_false`.
The declared output `taken_branch` used to render `{{ out_dir }}`, echoing an
input; it now reads a file the branch itself wrote, so it reports the branch.
Each example declares a case per branch, and a case now fixes the exit code,
every artifact *and its exact contents*, the complete outputs document, which
steps completed / were skipped / failed, and the execution levels -- so
"alpha and beta run in parallel" is a checked claim. An unexpected artifact is
a failure. `checkpoints/` is named as a run side effect and asserted directly,
rather than silently ignored.
CLI/API equivalence was nearly vacuous: the comparison dropped every step
value and the whole outputs document, on a comment claiming values embed
absolute paths. Measured across 5 examples x 3 runs, the only fields that ever
differ are the timestamps and the execution id; values are byte-identical and
paths in them are relative. `PipelineResult.normalized()` already blanked
exactly those, so `normalize_result_payload` now serves both the object and
the CLI's JSON and the test cannot drift into comparing less than it claims.
Two runtime defects surfaced from the stronger assertions:
A timed-out command was abandoned rather than killed. `asyncio.wait_for`
cancels the wait, not the process, and the timeout handler never referred to
the child again: 06_failure_policy left two `sleep` processes alive after the
orchestrator had exited. `_reap()` in a `finally` also covers the cancellation
raised when a step rather than the command times out, and closing the pipes on
a live loop is what stops the unraisable-exception warnings at teardown.
Templates dropped the trailing newline of any parameter referencing another
step, because that path alone goes through a Jinja environment left on the
default `keep_trailing_newline=False`. Literals and parameter references kept
theirs. Of 547 blocking tests the only two affected were the ones pinning the
truncated byte.
Every new test was run against the un-fixed code: the example cases fail 4x on
the old 04, the leak test names the surviving pid, the newline test fails on
the lost byte. The first draft of the leak test was vacuous -- its marker sat
in a shell comment, and `sh -c "sleep 30 # marker"` execs sleep in place and
drops it from argv -- so it now tracks process ancestry instead.
Blocking suite 541 -> 548 passed, 13 skipped, 0 failed. Unraisable and
resource warnings are errors in both affected suites.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CI verification9/9 green. Legacy tally against the last post-merge
The +7 deselected is exactly the 7 tests this PR adds to the blocking layer One thing I checked rather than assumedThe legacy job's annotations contain two It does not: the count is 2 on Worth filing separately, though. It is the same failure mode as the one this PR |
Responds to the review of #445. Covers its item 1 (repair the supported-example
contract) and item 5 (deeper assertions), plus item 3 (subprocess cleanup) and
one defect the stronger assertions surfaced.
The example was demonstrating the opposite of its claim
Confirmed before changing anything:
handle_shortwas never a branch. It was the last link of a dependency chain(
check_size -> handle_long -> handle_short), so the default path fell throughinto it.
on_falserouting worked; the example just did not express exclusivebranches — and
EXPECTATIONSasserted both artifacts, pinning it in place.Rewritten as two exclusive branches converging on a join. The long arm's
on_success: summarisejumps the short arm; the short arm is reached only byon_false. Measured, both arms:-i content=hi)taken_branch"long""short"taken_branchused to render{{ out_dir }}— it echoed an input and saidnothing about the branch. It now reads a file the branch itself wrote.
What a case asserts now
Exit code; every artifact and its exact contents, with an unexpected file
counting as a failure; the complete
outputsdocument; which steps completed,were skipped and failed; and the execution levels, so "alpha and beta run in
parallel" is checked rather than asserted in a comment.
checkpoints/is namedas a run side effect and asserted directly instead of silently ignored.
Examples with a branch declare a case per branch, so the arm a default run does
not take still executes in CI.
CLI/API equivalence was nearly vacuous
The comparison dropped every step
valueand the entireoutputsdocument, ona comment claiming values embed absolute paths differing by working directory.
Measured across all 5 examples x 3 repetitions, the only differing JSON paths
are
started_at,completed_at,duration,execution_id. Every step valueis byte-identical; paths inside them are relative (
./output/greeting.txt).The comment was wrong.
PipelineResult.normalized()already blanked exactly those fields and alreadyclaimed in its docstring to be the CLI/API comparison form, so this extracts
normalize_result_payload(dict)to serve both the object and the CLI's JSON.One definition — the test can no longer drift into comparing less than it says.
Two runtime defects the stronger assertions surfaced
A timed-out command was abandoned, not killed.
asyncio.wait_forcancelsthe wait, not the process, and the timeout handler never referred to the child
again:
_reap()in afinallyalso covers the cancellation raised when a steprather than the command times out. Closing pipes on a live loop is also what
stops the
PytestUnraisableExceptionWarningat teardown that #445's reviewreported.
Templates dropped a trailing newline — but only for a parameter referencing
another step:
"literal\n""param {{ word }}\n""step {{ reader.result.size }}\n"That path alone goes through a Jinja
Environmentleft on the defaultkeep_trailing_newline=False. Of 547 blocking tests, the only two affectedwere the ones pinning the truncated byte. Found because the suite now asserts
exact contents; the old suite asserted the file existed.
Evidence the new tests actually work
Every new test was run against the un-fixed code:
The leak test was vacuous in its first draft, and this is how I caught it:
the marker sat in a shell comment, and
sh -c "sleep 30 # marker"execssleepin place, dropping both the shell and the comment from argv. A renamed copy of
sleepwas then SIGKILLed by macOS code signing (rc=137). It now tracks processancestry with psutil.
Verification
PytestUnraisableExceptionWarningandResourceWarningare now errors intest_supported_examples.pyandtest_failure_policy.pycompileallclean; working tree cleanOn the review's "13 skips vs 12" note: skips depend on which optional extras are
installed, so that number is environment-dependent rather than a discrepancy.
Deliberately not in this PR
but it changes the execution loop, not the examples.
{"steps":..., "outputs":...}envelope(item 4).
docs/CLAUDE.mddirects that documentation describe the code as itis now and that migration guides not be written unless asked.
new examples respectively; neither belongs in a correctness change.
Examples validating is unchanged at 8 of 116 — this PR makes the supported set
trustworthy rather than larger.
🤖 Generated with Claude Code